[https://nvbugs/6758245][fix] Cherry-pick #20429: Keep /metrics from blocking the serving event loop - #20698

Description

Cherry-pick of #20429 (merged into release/1.3 as ae62a63) to main. Fixes https://nvbugs/6758245: turning enable_iter_perf_stats off made serving slower, because each GET /metrics scrape blocked the server's event loop on a stats read of up to 2 s.

• Stats off (commit by Bowen Fu, from [https://nvbugs/6758245][fix] Avoid metrics queue polling when iteration stats are disabled #19517): /metrics returns [] at once instead of waiting for stats that never arrive.
• Stats on without the background collector:
  • Dedicated thread: the stats read runs on a one-thread ThreadPoolExecutor, so streaming continues while it waits.
  • One read at a time: a scrape that arrives while a read is in flight gets [] at once instead of queueing a new read.
  • Cancelled scrapes: the stats a cancelled scrape's read drained go to the next scrape.
  • Shutdown: lifespan shuts the executor down after the generator.
• RPC failures: an RPCError from the read returns [] instead of a 500.
• Behavior change: each statistic is returned to only one caller. With two clients polling at once, one of them may get an empty list. The /metrics section of trtllm-serve.rst says so.

tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py is registered in l0_cpu.yml, as in #20429.

Conflict resolution: the only conflict was the import block of tensorrt_llm/serve/openai_server.py: main imports tensorrt_llm.executor.utils, which release/1.3 lacks, and this change adds tensorrt_llm.executor.rpc.RPCError. Both imports are kept. The change to openai_server.py is otherwise the same as in #20429 (63 changed lines in both).

Test Coverage

• tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py (CPU-Generic stages): all 13 tests pass locally on this branch, three runs in a row. They cover:
  • the stats-off early return;
  • event-loop responsiveness;
  • use of the metrics executor;
  • one read in flight under sustained scraping;
  • the cancellation and completion races;
  • RPCError handling.
• [https://nvbugs/6758245][fix] Keep /metrics from blocking the serving event loop #20429 passed the full release/1.3 pipeline, multi-GPU stages included (pipeline [Question]How do I check whether kv cache int8 is effective? #394).

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

Dev Engineer Review

/metrics now avoids blocking the serving event loop during iteration-stat reads. A single-thread executor prevents overlapping reads, preserves drained stats after cancellation, handles RPCError as an empty result, and shuts down during lifespan cleanup. No material API or configuration risks are identified.

QA Engineer Review

tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py covers slow reads, executor usage, disabled stats, concurrent reads, cancellation races, executor errors, and RPCError. The eager-start race test is restricted to Python 3.12 and newer. The test is listed in tests/integration/test_lists/test-db/l0_cpu.yml.

The author reports 13 passing tests across three local runs. A reported CI pipeline failed, and a later run has no completion result. Coverage verdict: needs follow-up.

Per-File QA Perspective

• tensorrt_llm/serve/openai_server.py: Verify non-blocking reads, single-read enforcement, cancellation reuse, executor shutdown, and RPCError handling.
• tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py: Covers the new metrics-read paths and failure cases. It is listed in the CPU CI test list.
• tests/integration/test_lists/test-db/l0_cpu.yml: Adds the iteration-stats test to CPU CI.
• docs/source/commands/trtllm-serve/trtllm-serve.rst: Documents different /metrics results for concurrent clients, including empty lists.